[Packaging] Bump embedded Python version to 3.12.7 #29887
Conversation
️✔️AzureCLI-FullTest
|
️✔️AzureCLI-BreakingChangeTest
|
|
Packaging |
bebound
changed the title
microsoft-github-policy-service
bot
requested review from
jiasli,
wangzelin007 and
yonzhan
bebound
changed the title
|
|
||
| echo "== Testing pip install on Python 3.11 ==" | ||
| docker run \ | ||
| --rm -v $PYPI_FILES:/mnt/pypi mcr.microsoft.com/mirror/docker/library/python:3.11-slim \ |
There was a problem hiding this comment.
MAR does not provide newer Python docker image anymore, use local Python instead.
Ref: https://mcr.microsoft.com/en-us/artifact/mar/mirror/docker/library/python/tags
| @@ -606,7 +597,7 @@ jobs: | |||
| -e CLI_VERSION=$CLI_VERSION \ | |||
| -e HOMEBREW_UPSTREAM_URL=$HOMEBREW_UPSTREAM_URL \ | |||
| --name azurecli \ | |||
| mcr.microsoft.com/mirror/docker/library/python:3.11-slim \ | |||
| mcr.microsoft.com/azurelinux/base/python:3 \ | |||
There was a problem hiding this comment.
Formula generation is independent of the Python version.
Use Azure Linux's 3.12 to avoid updates for the next four years.
docker source: https://mcr.microsoft.com/en-us/artifact/mar/azurelinux/base/python/tags
| shutil.rmtree(self._cache_dir, | ||
| ignore_errors=True, | ||
| onerror=lambda _, p, tr: self._logger.error('Fail to remove file %s', p)) | ||
| shutil.rmtree(self._cache_dir, ignore_errors=True) |
There was a problem hiding this comment.
onerror is only deprecated in 3.12, and is replaced by onexc. But there is no equivalent of onexc in earlier Python versions: python/cpython#112645.
onerror is meaningless when ignore_errors=True, so I remove it.
Ref: https://docs.python.org/3/library/shutil.html#shutil.rmtree
| source env/bin/activate | ||
| cd azure-cli-extensions | ||
| python ../scripts/ci/automation_full_test.py "12" "$(Instance_idx)" "latest" "" "True" "extension" | ||
| displayName: 'Rerun tests' |
There was a problem hiding this comment.
azdev can't find extension list when running in AzureCLI@1, move it to new step.
Traceback (most recent call last):
File "/mnt/vss/_work/1/s/azure-cli-extensions/../scripts/ci/automation_full_test.py", line 664, in <module>
extension_main()
File "/mnt/vss/_work/1/s/azure-cli-extensions/../scripts/ci/automation_full_test.py", line 654, in extension_main
autoscheduling.get_extension_modules()
File "/mnt/vss/_work/1/s/azure-cli-extensions/../scripts/ci/automation_full_test.py", line 546, in get_extension_modules
raise RuntimeError("No extension detected")
RuntimeError: No extension detected
There was a problem hiding this comment.
What is the error and do we know why?
There was a problem hiding this comment.
I think it's better not to change it until we figure out why.
bebound
modified the milestones:
September 2024 (2024-10-08),
October 2024 (2024-11-05)
bebound
changed the title
bebound
force-pushed
the
3.12
branch
3 times, most recently
from
afd296a to
f350858
Compare
bebound
force-pushed
the
3.12
branch
3 times, most recently
from
667728c to
d53b7dd
Compare
| @@ -24,7 +24,11 @@ exit_code=0 | |||
| # Disable k8s-extension temporarily: https://github.com/Azure/azure-cli-extensions/pull/6702 | |||
| # Disable alias temporarily: https://github.com/Azure/azure-cli/pull/27717 | |||
| # hybridaks is going to be deprecated: https://github.com/Azure/azure-cli/pull/29838 | |||
| ignore_list='azure-cli-ml fzf arcappliance arcdata connectedk8s k8s-extension alias hybridaks' | |||
| # db-up is going to be deprecated: https://github.com/Azure/azure-cli/pull/29887 | |||
| ignore_list='azure-cli-ml fzf arcappliance arcdata connectedk8s k8s-extension alias hybridaks db-up' | |||
There was a problem hiding this comment.
db-up's pymssql==2.2.7 does not provide wheel for 3.12. This extension is deprecated: Azure/azure-cli-extensions#8127
|
Our self-hosted agent provided by 1ES fails to build Windows package when using 3.12.7, but everything works fine in 3.12.6 embedded Python. When calling The microsoft-hosted agent works normally. The only difference I found is that it has Python 3.12.7. I suspect that calling commands without This issue exists for a long time, but only triggered in this PR. Here are some debug info: |
bebound
force-pushed
the
3.12
branch
8 times, most recently
from
3995731 to
ea25cd5
Compare
| REM Install wheel to force pip install azure-cli in legacy mode | ||
| REM see https://github.com/Azure/azure-cli/pull/29887 | ||
| echo Installing setuptools wheel | ||
| %PYTHON_DIR%\python.exe -Im pip install setuptools wheel |
There was a problem hiding this comment.
For setuptools:
Although setuptools is added to azure-cli dependency list in #27196, azure-cli is installed with --no-deps. Need to install it manually.
For wheel:
In 3.11, wheel is also installed when running get-pip.py. In 3.12, it's not installed. wheel is not included in final package in Linux and macOS. Actually, pip can still install wheel package without wheel. setuptools also removed wheel dependency in pypa/setuptools#1386.
However, wheel is still a factor for pip to use pep517 or not: https://github.com/pypa/pip/blob/ec5faeac4ef6bae97df0e779566ceb2b0de89d3f/src/pip/_internal/pyproject.py#L107 (ref: pypa/packaging.python.org#1517). If wheel is not installed, pip will use pep517 and fails, see #29887 (comment). The reason is that the -I param is missing in pip's build env. (Ref: pypa/pip#9081). See discussion in: pypa/pip#13023
So I also install wheel. (I think it's okay to remove wheel in the future)
# 3.11's get-pip.py
Successfully installed pip-24.2 setuptools-75.1.0 wheel-0.44.0
Pip set up successful
| # see https://github.com/Azure/azure-cli/pull/29887 | ||
| extra_dependencies = ['pip', 'setuptools'] | ||
| for dependency in extra_dependencies: | ||
| nodes[dependency] = research_package(dependency) |
There was a problem hiding this comment.
make_graph explicitly exclude pip and setuptools: https://github.com/tdsmith/homebrew-pypi-poet/blob/fdafc615bcd28f29bcbe90789f07cc26f97c3bbc/poet/poet.py#L131
Adding pip and setuptools in CLI's dependency does not work as brew's pip uses --no-deps option. Homebrew/homebrew-core#194884 (comment)
| # Does not exit if az extension add fails until all extensions have been tested | ||
| set +e |
There was a problem hiding this comment.
Will set +e make the script no longer fail in case of extension installation failure?
There was a problem hiding this comment.
This is similar to #14295 (comment).
There was a problem hiding this comment.
test_extensions.sh calls scripts/ci/artifacts.sh which calls scripts/ci/build.sh.
build.sh has set -e.
Once -e is set, this script fails immediately if one extension fails. set +e ensures all extension have been tested. Finally, line 59 returns exit_code.
| @@ -2,6 +2,8 @@ | |||
|
|
|||
| root=$(cd $(dirname $0); pwd) | |||
|
|
|||
| tdnf install -y ca-certificates | |||
There was a problem hiding this comment.
We switched the docker image from mcr.microsoft.com/mirror/docker/library/python:3.11-slim to mcr.microsoft.com/azurelinux/base/python:3 which doesn't intall ca-certificates by default.
Bump bundled Python to 3.12 and upgrade Python version in CI
The package size has also decreased, as the pyc file size is optimized in 3.12.
The windows zip package drops from 314MB to 299MB.
Related issue: #27957
Main changes:
-Iinbuild.cmdpipandsetuptoolsin homebrewHistory Notes
[Core] Resolve CVE-2024-6119
[Core] Resolve CVE-2024-5535
[Core] Resolve CVE-2024-4741
[Core] Resolve CVE-2024-4603
[Core] Resolve CVE-2024-2511